[don't merge yet] Pool QueueWithRetry and cache filterDirtyFiles regexp#19544
Open
AlexeyAkhunov wants to merge 1 commit intomainfrom
Open
[don't merge yet] Pool QueueWithRetry and cache filterDirtyFiles regexp#19544AlexeyAkhunov wants to merge 1 commit intomainfrom
AlexeyAkhunov wants to merge 1 commit intomainfrom
Conversation
AlexeyAkhunov
requested review from
AskAlexSharov,
mh0lt,
sudeepdino008 and
yperbasis
as code owners
AskAlexSharov
approved these changes
Mar 1, 2026
AlexeyAkhunov
force-pushed
the
pool-queue-cache-regexp
branch
2 times, most recently
from
aabd358 to
b6d591f
Compare
Contributor
Author
|
Looks like changes in this PR make the flaky test reproducible. I am using it to debug that flaky test |
AlexeyAkhunov
changed the title
…l executor lifecycle Three changes: 1. Pool QueueWithRetry via sync.Pool — avoids reallocating the 100K-element channel buffer on every block when ExperimentalBAL is enabled. Release() is idempotent (safe against double-call from deferred cleanup). 2. Cache compiled regexps in filterDirtyFiles via sync.Map. 3. Fix parallel executor cleanup ordering: wait for the exec loop goroutine to fully exit before stopping workers or releasing the queue. Without this, Release() can pool the queue while the exec loop is still calling Add(), leading to cross-test contamination. 4. Fix scheduleExecution: clear inProgress before re-pushing skipped tasks to pending (prevents phantom in-progress state). Add force-schedule fallback when all pending tasks are speculatively skipped to prevent deadlock. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
AlexeyAkhunov
force-pushed
the
pool-queue-cache-regexp
branch
from
f5d7e7d to
c0a4e11
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
execution/exec/txtask.go): Addsync.PoolforQueueWithRetrywith aRelease()method that drains the 100K-element channel without closing it, preserving the ~1.6MB buffer across reuses.parallelExecutor.runcallsRelease()instead ofClose(); workers exit viactx.Done(). Cleanup ordering ensuresstopWorkers()completes beforeRelease()to avoid races.db/state/dirty_files.go): Cache compiled regexps in async.Mapkeyed by pattern string. Each unique(filenameBase, ext)pair compiles once instead of on every call.Profiling testlab test runs showed QueueWithRetry allocation at 79.5GB (18.8% of total allocs) and filterDirtyFiles regexp at 14.6GB (3.3%). Combined expected savings: ~94GB (22% of total allocations).
Test plan
go build ./...passesgo test ./execution/exec/...andgo test ./execution/stagedsync/...andgo test ./db/state/...🤖 Generated with Claude Code